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Preface 


The  Institute  for  Defense  Analyses  (IDA)  prepared  this  paper  for  the  Office  of  the  Dep¬ 
uty  Under  Secretary  of  Defense  for  Science  and  Technology  (ODUSD(S&T))  under  the  “Tech¬ 
nology  Readiness  Assessments  and  Analyses”  task.  The  material  in  this  paper  was  presented  at 
the  National  Defense  Industrial  Association’s  9th  Annual  Systems  Engineering  Conference  in 
October  2006  and  will  be  presented  at  the  23rd  Annual  National  Test  and  Evaluation  Conference 
in  March  2007. 

This  paper  partially  fulfills  the  task  objective  of  developing  material  on  the  perfonnance 
and  review  of  technology  assessments  and  associated  processes.  The  author  would  like  to  thank 
Lance  Hancock  and  Lance  Roark  of  IDA  for  their  technical  reviews  of  this  paper. 
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Executive  Summary 


Thus  far,  Technology  Readiness  Assessment  (TRA)  guidance  has  primarily  focused  on 

(1)  hardware  and  software  critical  technology  elements  CTEs  that  affect  performance  and 

(2)  manufacturing-related  CTEs.  While  life-cycle -related  technologies  can  be  addressed  in  the 
current  TRA  process,  in  general,  they  receive  little  emphasis.  This  paper  describes  how  to 
increase  attention  on  such  technologies. 

A  TRA  is  a  regulatory  information  requirement  for  all  acquisition  programs.  It  is  a  sys¬ 
tematic,  metrics-based  process  that  uses  Technology  Readiness  Levels  (TRLs)  to  assess  the 
maturity  of  CTEs.  The  assessment  is  made  by  a  panel  of  subject  matter  experts  (SMEs)  inde¬ 
pendent  of  the  program.  The  Program  Manager  (PM),  Program  Executive  Officer  (PEO),  and 
Component  Acquisition  Executive  (CAE)  use  the  TRA  results  to  optimize  the  acquisition  strat¬ 
egy,  determine  the  capabilities  to  be  deferred  to  the  next  increment,  and  enhance  technology 
investment.  The  PM  also  uses  the  expertise  of  the  assessment  team  and  the  rigor  and  discipline 
of  the  process  for  an  early  in-depth  review  of  the  conceptual  product  baseline  and  periodic  in- 
depth  reviews  of  maturation  events.  The  TRA  also  highlights  (and,  in  some  cases,  discovers) 
critical  technologies  and  other  potential  technology  risk  areas  that  require  the  PM’s  attention 
(and  possibly  additional  resources).  In  addition,  the  Milestone  Decision  Authority  (MDA) — the 
single  focal  point  for  programmatic  decisions — uses  the  information  from  a  TRA  to  support  a 
decision  to  initiate  a  program  at  Milestone  B  and  then  later  to  support  a  decision  to  enter  Low- 
Rate  Initial  Production  (LRIP)  at  Milestone  C. 

While  any  CTE  has  an  effect  throughout  the  life  of  a  system,  this  paper  uses  the  tenn 
“life-cycle-related  technologies”  to  mean  those  technologies  that  affect  system  supportability 
cost  and/or  time.  They  can  reduce  the  logistics  footprint,  improve  reliability/maintainability, 
lower  operating  support  or  maintenance  manpower  requirements,  improve  training,  enhance 
human  factors  interactions,  increase  operational  availability  or  readiness,  or  improve  the  ability 
to  upgrade  of  the  system. 

To  improve  the  focus  on  life-cycle-related  technologies  during  CTE  identification, 
experts  in  the  life-cycle-related  areas  of  the  program  should  be  part  of  the  PM’s  technical  sup¬ 
port  team  that  makes  the  initial  CTE  detennination  and  of  the  independent  review  team,  which 
can  suggest  changes.  Also,  when  deciding  whether  a  CTE  candidate  is  critical,  additional  ques¬ 
tions  should  be  asked  to  ascertain  whether  a  candidate  life-cycle  technology  would  have  a  “sig¬ 
nificant”  affect  on  life-cycle  affordability: 

1 .  Is  the  affordability  benefit  significant  over  the  life  cycle,  where  significance  is  based 
on  the  judgment  of  the  independent  SME  panel? 

2.  Is  the  affordability  benefit  enabled  by  a  technological  solution  and  not  solely 
through  engineering  design? 
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These  questions  are  intended  to  highlight  whether  the  allocated  baseline  design  configuration 
contains  life-cycle-related  CTEs  and  must  be  answered  affirmatively. 

While  some  technologies  may  be  critical  solely  on  a  life-cycle  affordability  basis,  others 
may  be  critical  from  multiple  perspectives.  Consequently,  once  a  CTE  has  been  identified  from  a 
performance  perspective,  it  should  also  be  determined  whether  it  is  a  CTE  from  a  life-cycle  point 
of  view.  TRAs  evaluate  the  extent  to  which  a  program  is  ready  to  transition  to  the  next  phase  of 
development — an  evaluation  that  is  based  on  the  maturity  of  the  critical  technologies.  Therefore, 
if  a  technology  is  critical  from  both  perspectives  (perfonnance-related  maturity  and  life-cycle- 
related  maturity),  it  should  be  assessed  from  both  perspectives.  Establishing  performance-related 
maturity  is  not  a  sufficient  condition  for  assuming  life-cycle-related  maturity.  Separate  TRLs 
should  be  assigned. 

CTE  maturation  should  be  monitored  throughout  the  System  Development  and  Demon¬ 
stration  (SDD)  Phase  of  the  acquisition  framework.  All  CTEs  should  have  a  maturation  plan  that 
shows  a  roadmap  for  reaching  TRL  8  (actual  system  proven  through  successful  mission  opera¬ 
tions).  An  independent  technical  authority  should  monitor  the  status  of  CTE  maturation  plans 
during  systems  engineering  technical  reviews.  In  addition,  these  technical  reviews  will  be  the 
forum  for  identifying  any  new  life-cycle-related  CTEs  that  emerge  as  part  of  the  solution  to  a 
problem  encountered  during  system  development. 
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Identifying  and  Assessing  Life-Cycle-Related 
Critical  Technology  Elements  (CTEs) 
for  Technology  Readiness  Assessments  (TRAs) 


A.  Background 

A  Technology  Readiness  Assessment  (TRA)  is  a  regulatory  information  requirement  for 
all  acquisition  programs.  It  is  a  systematic,  metrics-based  process  that  uses  Technology  Readi¬ 
ness  Levels  (TRLs)  to  assess  the  maturity  of  critical  technology  elements  (CTEs).  The  assess¬ 
ment  is  made  by  a  panel  of  subject  matter  experts  (SMEs)  independent  of  the  program.  A 
summary  description  of  CTEs  and  TRLs  from  the  TRA  Deskbook\  is  as  follows: 

•  A  technology  element  is  “critical”  if  the  system  being  acquired  depends  on  this  tech¬ 
nology  element  to  meet  operational  requirements  with  acceptable  development  cost 
and  schedule  and  with  acceptable  production  and  operation  costs  and  if  the  technol¬ 
ogy  element  or  its  application  is  either  new  or  novel. 

•  TRLs  indicate  what  has  been  accomplished  in  the  development  of  a  technology  from 
several  perspectives:  theory  to  laboratory  to  field,  relevant  environment  to  opera¬ 
tional  environment,  subscale  to  full  scale,  breadboard  to  brassboard  to  prototype,  and 
partial  performance  to  full  performance.  TRLs  do  not  indicate  that  the  technology  is 
right  for  the  job  or  that  the  application  of  the  technology  will  result  in  successful 
development  of  the  system,  and  TRLs  do  not  address  risk  or  system  integration.1 2 

Many  programs  that  begin  development  with  immature  technologies  have  experienced 
significant  technical  difficulties,  which  lead  to  schedule  delays  and  cost  overruns.  The  TRA  was 
established  as  a  control  mechanism  to  identify  and  monitor  the  maturity  of  critical  technologies, 
based  on  what  has  been  accomplished.  The  Milestone  Decision  Authority  (MDA) — the  single 
focal  point  for  programmatic  decisions — uses  the  information  from  a  TRA  to  support  a  decision 
to  initiate  a  program  at  Milestone  B  and  then  later  to  support  a  decision  to  enter  Low-Rate  Initial 
Production  (LRIP)  at  Milestone  C.  Congress  has  recognized  the  relationship  between  program 
success  and  TRAs.  At  program  initiation,  the  MDA  must  certify  to  Congress  that  the  technology 


1  Department  of  Defense  Technology  Readiness  Assessment  (TRA)  Deskbook,  prepared  by  the  Deputy  Under 
Secretary  of  Defense  for  Science  and  Technology  (DUSD(S&T)),  May  2005, 
http://www.defenselink.mil/ddre/weapons.htm. 

2  The  TRA  Deskbook  addresses  hardware-,  software-,  and  manufacturing-related  technologies.  The  TRL 
definitions  for  the  hardware-  and  manufacturing-related  technologies  are  identical  but  are  different  from  the 
TRL  definitions  for  the  software-related  technologies.  This  paper  addresses  only  hardware-  and  software- 
related  technologies. 
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in  Major  Defense  Acquisition  Programs  (MDAPs)  has  been  demonstrated  in  a  relevant  environ¬ 
ment.3  Waivers  of  this  certification  for  national  security  must  be  justified.4 

The  Program  Manager  (PM),  Program  Executive  Officer  (PEO),  and  Component  Acqui¬ 
sition  Executive  (CAE)  use  the  TRA  results  to  optimize  the  acquisition  strategy,  determine  the 
capabilities  to  be  deferred  to  the  next  increment,  and  enhance  technology  investment.  In  addi¬ 
tion,  the  PM  uses  the  expertise  of  the  independent  assessment  team  (i.e.,  the  SMEs)  and  the  rigor 
and  discipline  of  the  process  for  an  early  in-depth  review  of  the  conceptual  product  baseline  and 
periodic  in-depth  reviews  of  maturation  events.  The  TRA  also  highlights  (and,  in  some  cases, 
discovers)  critical  technologies  and  other  potential  technology  risk  areas  that  require  the  PM’s 
attention  (and  possibly  additional  resources). 

Thus  far,  TRA  guidance  has  focused  primarily  on  (1)  hardware  and  software  CTEs  that 
affect  performance  and  (2)  manufacturing-related  CTEs.  While  life-cycle-related  technologies 
can  be  addressed  in  the  current  TRA  process,  in  general,  they  receive  little  emphasis.  This  paper 
describes  how  to  increase  attention  on  such  technologies. 

B.  Significance  of  Life-Cycle-Related  Technologies  to  TRAs 

While  any  CTE  has  an  effect  throughout  the  life  of  a  system,  this  paper  uses  the  term 
“life-cycle-related  technologies”  to  mean  those  technologies  that  affect  system  supportability 
cost  and/or  time.  They  can  reduce  the  logistics  footprint,  improve  reliability/maintainability, 
lower  operating  support  or  maintenance  manpower  requirements,  improve  training,  enhance 
human  factors  interactions,  increase  operational  availability  or  readiness,  or  improve  the  ability 
to  upgrade  of  the  system. 

Examples  of  life-cycle-related  technologies  include 

•  Corrosion-resistant  materials 

•  Thermal  protection  materials 

•  Supportable  low-observable  (LO)  materials 

•  Obsolescence  mitigation  technologies 

•  Technical  data  automation  technologies 

•  Material  handling  technologies 

•  Simulators  or  training  simulations 


3  Section  2366a  of  Title  10,  United  States  Code  (U.S.C.),  as  enacted  by  Section  801  of  the  National  Defense 
Authorization  Act  (NDAA)  for  Fiscal  Year  (FY)  2006  (P.L.  109-163).  [Editor’s  note:  The  Public  Law  (P.L.) 
number  follows  the  form  P.L.  109-163,  meaning  this  law  is  the  163rd  law  passed  by  the  109th  Congress .] 

4  The  MDA  can  waive  any  certification  requirement  of  P.L.  109-163  (as  enacted  by  Section  801)  if  the  Depar¬ 
tment  is  unable  to  meet  national  security  objectives.  The  MDA  has  to  submit  the  waiver,  the  determination,  and 
reasons  for  the  determination,  in  writing,  to  the  congressional  defense  committees  within  30  days  of  authorizing 
the  waiver. 
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•  Autonomic5  logistics  sensors,  data  links,  or  messaging  transmission 

•  Advanced  technologies  that  affect  human  factors  such  as  safety  and  occupational 
health,  habitability,  or  cognitive  or  physical  requirements 

•  Analysis  technologies,  such  as  automated  diagnostics  and  prognostics 

•  Methods/algorithms  for  sensing  or  trend  analysis 

•  Technologies  that  enable  open  systems  architectures. 

The  TRA  process  should  be  concerned  with  such  technologies  for  many  reasons.  First, 
from  a  definitional  perspective,  the  CTE  definition  encompasses  all  elements  of  the  life  cycle  in 
tenns  of  development,  production,  and  operating  and  support  (O&S)  costs. 

Second,  from  a  policy  perspective,  life-cycle-related  issues  affect  military  capability,  and, 
therefore,  greater  emphasis  is  being  placed  on  the  technologies  that  enable  this  capability. 
Department  of  Defense  Instruction  (DoDI)  5 000. 2 6  states  that  “The  project  shall  exit 
Technology  Development  when  an  affordable  increment  of  militarily-useful  capability  has  been 
identified,  the  technology  for  that  increment  has  been  demonstrated  in  a  relevant  environment, 
and  a  system  can  be  developed  for  production  within  a  short  timeframe  (normally  less  than  5 
years);  or  when  the  MDA  decides  to  tenninate  the  effort.”  Chainnan  of  the  Joint  Chiefs  of  Staff 
Instruction  (CJCSI)  3170.01E7  defines  increment  as  “a  militarily  useful  and  supportable 
operational  capability  that  can  be  effectively  developed,  produced  or  acquired,  deployed,  and 
sustained.  Each  increment  of  capability  will  have  its  own  set  of  threshold  and  objective  values 
set  by  the  user.  Spiral  development  is  an  instance  of  an  incremental  development  strategy  where 
the  end  state  is  not  known.  Technology  is  spiraled  to  maturity  and  injected  into  the  delivery  of  an 
increment  of  capability.”  Accordingly,  mobility  and  logistics  footprint  are  military  capabilities 
and  reliability  and  maintainability  are  military  performance  parameters. 

Finally,  from  a  real-life  experience  perspective,  O&S  costs  will  continue  to  increase, 
thereby  making  life-cycle-related  technologies  even  more  critical.  The  percentage  of  systems 
passing  operational  tests  from  a  suitability8  perspective9  has  experienced  a  recent  drop  (see 


5  Autonomic  refers  the  ability  respond  to  problems,  repair  faults,  and  recover  from  system  outages  without  the 
need  for  human  intervention. 

6  DoDI  5000.2,  Operation  of  the  Defense  Acquisition  System,  May  12,  2003. 

7  CJCSI  3 170.0  IE,  Joint  Capabilities  Integration  and  Development  System,  May  11,  2005. 

8  According  to  CJCSI  3170.01E,  operational  suitability  is  the  degree  to  which  a  system  can  be  placed  and 
sustained  satisfactorily  in  field  use,  with  consideration  given  to  availability;  compatibility;  transportability; 
interoperability;  reliability;  wartime  usage  rates;  maintainability;  environmental,  safety,  and  occupational  health 
risks;  human  factors;  habitability;  manpower;  logistics  supportability;  natural  environment  effects  and  impacts; 
and  documentation  and  training  requirements. 

9  The  trend  has  been  that  more  systems  are  passing  operational  tests  from  an  effectiveness  perspective.  This  trend 
is  primarily  the  result  of  a  change  in  testing  philosophy.  Previously,  most  tests  were  conducted  on  a  pass-fail 
basis  against  a  specific  (and  sometimes  arbitrary)  required  performance  value  or  level.  In  today’s  environment, 
testing  is  based  on  the  ability  to  accomplish  the  mission. 
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Figure  1).  Interviews  with  several  SMEs  attribute  this  drop  to  an  overemphasis  on  performance 
by  program  management. 
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Figure  1.  Percentage  of  Systems  Passing  an  Operational  Test 

Resources  for  suitability  are  often  diverted  to  deal  with  technical  performance  problems, 
and  this  means  that  technical  suitability  issues  have  to  be  solved  after  production  when  such 
solutions  will  prove  more  costly  and  cause  delays  to  a  program.  Recently,  an  abundance  of  soft¬ 
ware  development  problems  have  exacerbated  these  diversions. 

C.  Life-Cycle  CTE  Identification 

The  TRA  Deskbook  describes  a  three  step  process  used  to  identify  CTEs: 

•  Step  1.  Use  the  work  breakdown  structure  (WBS)  or  system  architecture  for  infor¬ 
mation  technology  systems  to  identify  CTE  candidates  by 

-  Establishing  the  functions  to  be  performed  by  each  system,  subsystem,  or  com¬ 
ponent  throughout  the  WBS,  determining  how  these  functions  will  be  accom¬ 
plished,  and  identifying  the  technologies  needed  to  perfonn  these  functions  at 
the  desired  level. 

•  Step  2.  Determine  whether  the  candidate  technology  is  critical  to  the  program  by 
answering  one  of  the  following  questions  affirmatively: 

-  Does  the  technology  directly  affect  an  operational  requirement? 

-  Does  the  technology  significantly  effect  an  improved  delivery  schedule? 

-  Does  the  technology  significantly  affect  the  affordability  of  the  system? 

•  Step  3.  Determine  whether  the  candidate  technology  is  new  or  novel  by  answering 
one  of  the  following  questions  affirmatively: 

-  Is  the  technology  new  or  novel? 

-  Has  the  technology  been  modified? 
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-  Has  the  technology  been  repackaged  such  that  a  new,  relevant  environment  is 
realized? 

-  Is  the  technology  expected  to  operate  in  an  environment  and/or  achieve  a  level 
of  performance  beyond  its  original  design  intention  or  demonstrated  capability? 

These  steps  are  normally  conducted  by  the  PM  (and  his/her  technical  support  staff).  As  a 
best  practice,  the  independent  panel  of  SMEs  who  make  the  readiness  assessments  for  the  tech¬ 
nology  should  also  be  asked  to  verify  the  PM’s  initial  CTE  list  and  determine  whether  additions 
or  deletions  are  warranted.  Expanding  the  third  question  in  step  2  could  increase  the  likelihood 
of  identifying  life-cycle-related  technologies.  Step  2  would  become: 

•  Step  2  revised.  Determine  whether  the  candidate  technology  is  critical  to  the  pro¬ 
gram  by  answering  one  of  the  following  questions  affirmatively: 

-  Does  the  technology  directly  effect  an  operational  requirement? 

-  Does  the  technology  have  a  significant  effect  on  an  improved  delivery 
schedule? 

-  Does  the  technology  have  a  significant  effect  on  the  life-cycle  affordability  of 
the  system? 

This  revised  question  can  be  answered  affirmatively  from  an  acquisition  cost  perspective 
with  a  “yes”  answer  to  the  following  amplifying  questions: 

•  Will  the  acquisition  cost  of  the  component  or  subsystem  that  uses  this  technology  be 
significantly  higher  without  the  technology? 

The  revised  question  can  also  be  answered  affirmatively  from  an  O&S  cost  structure  per¬ 
spective  with  a  “yes”  answer  to  any  of  the  following  amplifying  questions: 

•  Does  the  technology  significantly  reduce  the  logistics  footprint? 

•  Does  the  technology  significantly  improve  reliability  or  maintainability? 

•  Does  the  technology  significantly  lower  operational,  support,  or  maintenance  man¬ 
power  requirements? 

•  Does  the  technology  significantly  improve  training  by  some  combination  of  low¬ 
ering  the  resources  needed  for  training  or  boosting  the  effectiveness  of  the  training? 

•  Does  the  technology  significantly  increase  operational  availability  or  readiness? 

•  Does  the  technology  significantly  improve  the  ability  to  upgrade  the  system? 

These  amplifying  questions  are  not  intended  to  suggest  that  technologies  not  encom¬ 
passed  in  the  CTE  definition  should  be  assessed  in  a  TRA.  They  are,  however,  intended  to  high¬ 
light  whether  the  allocated  baseline  design  configuration  contains  life-cycle-related  CTEs.  Two 
conditions  must  be  met  to  answer  affirmatively: 

1 .  The  affordability  benefit  must  be  significant  over  the  life  cycle,  where  significance  is 
based  on  the  judgment  of  the  independent  SME  panel. 


5 


2.  The  affordability  benefit  must  be  enabled  by  a  technological  solution  and  not  solely 
through  engineering  design. 

In  some  cases,  a  performance-related  CTE  can  also  be  designated  as  a  life-cycle-related 
CTE  (see  Figure  2).  The  technology  would  have  to  be  “new  or  novel”  and  would  have  to  affect 
an  operational  requirement  and  significantly  affect  the  life-cycle  affordability  of  the  system.  The 
TRA’s  purpose  is  to  determine  whether  critical  technologies  are  mature  enough  for  the  program 
to  enter  the  next  phase  of  development.  Since  maturity  from  a  performance  perspective  does  not 
necessarily  imply  maturity  from  a  life-cycle-related  perspective,  technologies  identified  as  criti¬ 
cal  from  two  perspectives  should,  therefore,  be  assessed  from  both  perspectives  in  the  TRA. 


•  Does  the  technology  directly  affect  an  operational  requirement? 

Yes.  The  performance  and  capabilities  of  current  airborne  radars  are 
limited  by  the  speed  of  the  mechanically  scanned  antennas.  The 
APG-79’s  beam  can  be  steered  close  to  the  speed  of  light,  thereby 
enabling  superior  performance,  including  air-to-air  tracking  at  very 
long  detection  ranges — almost  simultaneous  air-to-air  and  air-to- 
surface  mode  capability — and  improved  situational  awareness. 

•  Does  the  technology  have  a  significant  affect  on  the  life-cycle 
affordability  of  the  system?  Yes.  The  technology  enables  a  modular  design,  open  systems 
architecture  that  leads  to  rapid  repairs  and  the  ability  to  be  upgrded  easily.  Because  the  array  is 
solid  state,  mechanical  breakdowns  are  virtually  eliminated.  Its  predicted  mean  time  between 
critical  failures  (MTBCFs)  is  greater  than  15,000  hours.  The  MTBCF  for  the  entire  system  will 
be  greater  than  1,250  hours. 

•  Is  the  technology  new  or  novel?  Yes.  Everything  in  the  system  is  new — from  front-end  array 
to  back-end  processor  and  operational  software. 

Consequently,  the  active  electronically  scanned  array  (AESA)  radar  is  a  CTE  from 
performance  and  life-cycle  affordability  viewpoints. 

Figure  2.  Illustrative  Example  of  a  CTE  That  Is  Both  Performance  Related  and  Life  Cycle  Related: 
the  AN/APG-79  AESA  Radar,  a  Key  Element  in  the  F/A-18  E/F  Super  Hornet  Block  II  Upgrades10 

D.  Assessing  the  Maturity  of  Life-Cycle-Related  CTEs 

The  TRL  tables  in  the  TRA  Deskbook  were  designed  to  assess  the  maturity  of  hardware-, 
software-,  and  manufacturing-related  technologies.  The  columns  in  these  tables  provide 


10  Sources:  All  source  material  copyright  Raytheon  Company  -  Rights  reserved  under  copyright  laws  of  the  United  States.  Permission  is 
granted  by  Raytheon  Company  for  the  U.S.  Government  to  copy  this  material  for  evaluation  purposes  only. 

(1)  June  28,  2005,  Raytheon  News  Release,  Raytheon’s  Revolutionary  APG-79  AESA  Radar  is  Awarded  a  $580 Million  Multi-Year 
Procurement  Contract  by  the  Boeing  Company, 

http://www.pmewswire.com/cgi-bin/micro  stories.pl? ACCT=1 49999&TICK=RTN&STORY=/www/story/06-28- 

2005/0003985043&EDATE=Jun+28.+2005. 

(2)  November  20,  2002,  Raytheon  News  Release,  Raytheon  Demonstrates  First  Next-Generation  AESA  Capability  at  APG-79  Event, 
http://www.ravtheon.eom/newsroom/briefs/l  12002.htm. 

(3)  Raytheon  AN/APG-79  AESA  Data  Sheet, 

http://www.raytheon.com/products/stellent/groups/sas/documents/legaey  site/cmsOl  05083 1  .pdf. 
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definitions,  descriptions,  and  supporting  information  for  the  applicable  TRLs.  The  “supporting 
information  column”  is  a  brief  explanation  of  some  specific  criteria  to  apply  (sufficient  condi¬ 
tions)  when  assigning  a  TRL  to  a  technology. 

All  the  information  in  these  tables  applies  to  life-cycle-related  CTEs.  An  assessment 
should  use  either  the  software  or  the  hardware  definitions  and  descriptions  verbatim,  depending 
on  the  nature  of  the  technology.  However,  assessing  the  maturity  of  life-cycle-related  CTEs  can 
be  aided  by  supplementing  the  “supporting  information  column”  with  some  clarifying  examples 
tailored  to  such  technologies.  The  primary  reason  is  this:  Beyond  TRL  3  (analytical  and  experi¬ 
mental  critical  function  and/or  characteristic  proof  of  concept),  the  long-term  effects  of  life- 
cycle-related  technologies  must  be  demonstrated  in  a  short  period  of  time. 

Under  circumstances  in  which  the  long-term  effects  of  the  life-cycle-related  CTE  can  be 
calculated  analytically  and  the  risk  of  error  is  minimal, 1 1  the  existing  supporting  information  (for 
TRL  4  and  above)  is  sufficient  (e.g.,  technologies  that  reduce  manpower  O&S  requirements 
through  automation,  enhance  training,  or  improve  the  ability  of  the  system  to  be  upgraded)  (see 
Figures  3  and  4). 

When  long-tenn  effects  cannot  be  accurately  calculated  analytically  and  the  risk  of  a 
miscalculation  is  large  (e.g.,  accelerated  life  testing  of  explosives),  the  usual  supporting  infor¬ 
mation  for  TRL  4  and  above  should  be  augmented  or  tailored  for  the  specific  situation.  For 
example, 

•  For  technologies  that  improve  reliability/maintainability  and  correspondingly  reduce 
the  logistics  footprint  and  operating,  support,  or  maintenance  manpower  require- 
ments,  the  additional  supporting  infonnation  should  focus  on  the  performance  of 
the  end  item  in  these  areas  (see  Figure  5). 

•  For  technologies  used  to  protect  against  the  environment,13  the  additional  supporting 
information  should  focus  on  the  performance  of  the  material  being  tested  (see  Fig¬ 
ure  6). 

•  For  on-board  and  off-board  technologies  for  analysis,  status,  or  diagnosis  of  failure, 
the  additional  supporting  information  should  focus  on  accuracy  (see  Figure  7). 

Tables  1-6  suggest  additional  supporting  information  for  the  hardware  TRL  data  for 
technologies  used  to  improve  reliability/maintainability  and  to  protect  against  the  environment. 
These  tables  also  suggest  additional  supporting  infonnation  for  the  software  TRL  data  for  analy¬ 
sis  technologies.  In  these  situations,  the  technologies  are  often  based  on  commercial-off-the- 
shelf  (COTS)  products  or  on  similar  applications  in  other  Department  of  Defense  (DoD)  sys¬ 
tems.  Therefore,  the  necessary  supporting  information,  as  described  in  the  tables,  might  be 
obtained  from  such  sources. 


1 1  Risk  is  associated  with  the  calculations  themselves  and  with  the  effects  of  errors  in  the  calculations. 

13  Includes  redesigns  either  for  reliability  improvements  or  because  of  obsolescence  mitigation.  Often,  a  form-fit- 
function  replacement  for  an  existing  item  but  may  also  include  increased  functionality. 

13  Such  technologies  also  often  protect  the  environment. 
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Description:  The  CVN-21  program  is  designing  the  aircraft  carrier 
for  the  21st  Century,  as  the  replacement  for  the  Nimitz  Class  nuclear 
aircraft  carriers.  Improved  weapons  handling  is  one  of  the  new 
features  of  this  ship  class.  The  proposed  shipboard  weapons  loader 
(SWL)  combines  human  amplification  technology  with  a  self- 
powered  platform,  high-torque  electric  actuator/motors,  and  variable 
geometry  ilonator  wheels.  This  will  provide  a  capability  for  a  single 
munitions  while  reducing  operator  workload  and  life-cycle  cost, 
classified  as  a  life-cycle-related  CTE. 14 

Example  of  an  Assessment:  TRL  4  (component  and/or  breadboard  validation  in  a  laboratory  environ¬ 
ment)  was  achieved  in  successful  demonstrations  of  the  SWL  in  shore-based  industrial  environments 
with  a  single  human  operator.  Ship  manpower  reductions  were  confirmed  across  the  entire  fleet  through 
a  comparison  with  current  SWL  crew  requirements.  Planned  demonstration  of  shipboard  prototypes 
with  human  system  controls  and  ship’s  motion  compensation  under  relevant  shipboard  environments 
would  be  required  to  achieve  TRL  6. 


Figure  3.  Illustrative  Example  of  a  Life-Cycle-Related  CTE  Where  Existing  TRL  Supporting 
Information  Is  Sufficient  for  Assessing  Maturity:  the  SWL  for  the  CVN-21 


Description:  As  the  world’s  only  fifth-generation  fighter,  the  F-22  Raptor 
is  being  developed  to  counter  increasingly  sophisticated  air  and  ground 
threats  that  the  F-15  cannot  readily  defeat.  The  F-22’s  requirements 
emphasize  the  reliability  and  maintainability  of  systems.  By  replacing 
paper-based  technical  orders  (TOs),  the  new  Integrated  Management 
Information  System  (IMIS)  increases  the  accuracy  of  technical  data, 
accelerates  the  preparation  of  work  orders  and  parts’  requisitions,  and 
improves  the  performance  of  maintenance  specialists  and  technicians.  IMIS  could,  therefore,  be  classi¬ 
fied  as  a  life-cycle-related  CTE. 15 

Example  of  an  Assessment:  TRL  6  (system/subsystem  model  or  prototype  demonstration  in  a  relevant 
environment)  was  accomplished  in  a  field  test  of  a  prototype  system  that  was  near  the  desired  configu¬ 
ration.  The  tests  were  performed  on  three  F-16  subsystems.  Test  results  demonstrated  across-the-board 
performance  improvements,  as  measured  by  reduced  error  rates  and  fewer  maintenance  man-hours.16 
These  results  can  be  extrapolated  to  the  entire  F-22  fleet  with  little  risk  of  error. 


Figure  4.  Illustrative  Example  of  a  Life-Cycle  Related  CTE  Where  Existing  TRL  Supporting 
Information  Is  Sufficient  for  Assessing  Maturity:  the  IMIS  for  the  F-22 


14  The  SWL  is  a  CTE  for  the  CVN-21.  It  was  rated  TRL  4  in  the  TRA. 

15  No  TRA  has  been  done  for  the  F-22. 

16  For  more  information,  see  IDA  Paper  P-3173,  Costs  and  Benefits  of  the  Integrated  Maintenance  Information 
System  (IMIS),  May  1996. 
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Description:  The  C-5  Galaxy  is  a  heavy  cargo  transport  plane 
designed  to  provide  strategic  airlift  for  the  deployment  and 
supply  of  forces.  The  C-5  Reliability  Enhancement  and 
Re-Engining  Program  (RERP)  is  a  comprehensive  effort  to 
improve  reliability,  maintainability,  and  availability.  While 
this  program  encompasses  the  upgrade  of  more  than  50  items, 
the  greatest  expected  benefit  will  be  achieved  by  replacing  the 
current  engine  with  a  new  engine  (along  with  new  pylons, 
wing  attachment  fittings,  software,  and  thrust  reversers).  Therefore,  the  new  propulsion  system  can  be 
the  source  of  several  life-cycle-related  CTEs. 

Example  of  an  Assessment: 1 7  TRL  7  (system  prototype  demonstration  in  an  operational  environment) 
was  accomplished  because  of  commercial  experience  with  the  General  Electric  CF-680C2  engine  being 
used  to  replace  the  current  TF39  engine.  The  increase  of  engine  hardware  reliability  to  more  than 
10,000  hours  of  engine  time  on  wing  has  been  demonstrated  from  commercial  experience.  Commercial 
data  have  also  established  “fix  rates”  as  follows:  at  least  30.1  percent  of  failures  will  be  corrected 
within  4  hours,  at  least  62.9  percent  of  failures  will  be  corrected  within  12  hours,  and  at  least 
82.4  percent  of  failures  will  be  corrected  within  24  hours.  These  fix  rates  will  enable  a  significant 
increase  in  sortie  generation.  One  difference  from  the  commercial  version  is  that  new  pylons,  wing 
attachment  fittings,  software,  and  thrust  reversers  will  be  used  on  the  C-5.  The  new  pylons  and  wing 
attachment  fittings  have  been  employed  successfully  in  similar  applications  and  operational  environ¬ 
ments.  Most  of  the  basic  algorithms  in  the  engine  software  will  not  change.  While  the  new  engine 
thrust  reverser  is  COTS,  the  C-5  is  required  to  use  its  thrust  reverser  in-flight  for  rapid  descent  capabil¬ 
ity — a  maneuver  that  is  not  made  in  commercial  applications. 


Figure  5.  Illustrative  Example  of  Assessing  the  Maturity  of  Life-Cycle-Related  CTEs 
To  Improve  Reliability/Maintainability:  the  C-5  RERP 


*7  The  C-5  TRA  identified  the  components  discussed  herein  as  CTEs.  The  TRL  assessment  in  the  actual  TRA 
differed  from  this  example. 
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Description:  The  new  Landing  Helicopter  Assault  Replacement 
(LHA-R)  amphibious  assault  ship  is  slated  to  replace  LHA-1  Class 
ships.  Because  the  LHA-R  will  be  an  enhanced  aviation  variant  of 
the  LHD-8,  redesign  plans  were  developed  for  the  affected  systems. 

There  is,  however,  an  opportunity  to  reduce  life-cycle  costs 
significantly  by  using  advanced  high  solid,  low-edge  retentive  tank 
coatings  instead  of  solvent-based  paints.  In  the  last  decade,  the 
represervation  of  tanks  has  represented  the  highest  annual 
maintenance  cost  to  the  fleet.  These  new  advances  could  be  classified  as  a  life-cycle-related  CTE.18 

Example  of  an  Assessment:  An  abundance  of  commercial  data  is  available  on  high  solid,  low-edge 
retentive  coatings.  These  new  coatings  have  an  estimated  service  life  of  20  years  as  compared  with  the 
2-  to  7-year  service  life  of  the  current  coatings.  Savings  are  realized  from  reduced  inspection,  cleaning, 
preparation,  and  painting  requirements.  In  addition,  applying  the  new  coatings  is  less  labor  intensive 
because  fewer  coats  are  needed.  Overall,  a  TRL  7  (system  prototype  demonstration  in  an  operational 
environment)  can  be  assigned  to  this  CTE. 


Figure  6.  Illustrative  Example  of  Assessing  the  Maturity  of  Life-Cycle-Related 
Environmental  Protection  CTEs:  High  Solid,  Low-Edge  Retentive  Tank  Coatings 
for  the  LHA-R  Amphibious  Assault  Ship 


Description:  The  extensive  use  of  predictive  maintenance, 
conducted  by  networked  on-board  diagnostics  and  prognostics 
that  pulse  the  system  when  issues  arise  (or  are  expected),  is  an 
important  component  of  the  Joint  Integrating  Concept  for  Joint 
Logistics.  Such  anticipatory  knowledge  provides  commanders 
important  advantages  for  successfully  achieving  the  mission.  The 
Future  Combat  Systems  (FCS),  therefore,  plans  to  use  embedded 
predictive  logistics  sensors  and  algorithms  and  could  classify 
them  as  life-cycle-related  CTEs.19 

Example  Assessment:  When  failures  are  random,  physics-of-failure  models  do  not  exist.  Conse¬ 
quently,  a  statistical  approach  to  prediction  must  be  taken;  however,  currently,  no  data  are  available  to 
support  such  an  approach  for  the  FCS.  In  the  electronics  area,  some  work  is  underway  to  model  degra¬ 
dations  as  a  result  of  vibrations  and  thermal  cycles.  This  work,  however,  is  relatively  immature — only 
TRL  4  [module  and/or  subsystem  validation  in  a  laboratory  environment  (i.e.,  software  prototype 
development  environment)]  is  applicable.  The  Army  is  also  deploying  a  health  monitoring  system  for 
the  Patriot  Advanced  Capability  (PAC)  missile  system.  Thus,  while  some  FCS  mission-critical  systems 
may  be  TRL  5  (module  and/or  subsystem  validation  in  a  relevant  environment),  most  are  TRL  4,  and, 
overall,  an  embedded  predictive  logistics  sensors  and  algorithms  CTE  would  be  TRL  4. 
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Figure  7.  Illustrative  Example  of  Assessing  the  Maturity  of  Life-Cycle-Related  Analysis  CTEs: 
Embedded  Predictive  Logistics  Sensors  and  Algorithms  for  the  Army’s  FCS 


^  The  LHA-R  TRA  identified  no  CTEs.  However,  the  Defense  Acquisition  Board  (DAB)  was  made  aware  of  an 
issue  concerning  edge  retentive  tank  coating  opportunities.  A  business-case  analysis  was  requested. 
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Table  1.  Additional  Life-Cycle-Related  Supporting  Information  for  TRL  4 


Hardware  TRL  4 
Definition 

Hardware  TRL  4 

Description 

Hardware  TRL  4 
Supporting  Information 

Component  and/or 
breadboard  validation 
in  a  laboratory 
environment. 

Basic  technological  components  are  integrated  to 
establish  that  they  will  work  together.  This  is  rela¬ 
tively  “low  fidelity”  compared  with  the  eventual 
system.  Examples  include  integration  of  “ad  hoc” 
hardware  in  the  laboratory. 

System  concepts  that  have  been 
considered  and  results  from  testing 
laboratory-scale  breadboard(s).  Ref¬ 
erences  to  who  did  this  work  and 
when.  Provide  an  estimate  of  how 
breadboard  hardware  and  test 
results  differ  from  the  expected  sys¬ 
tem  goals. 

Additional  hardware  supporting  information 
for  technologies  to  improve  reliability/maintainability 

Analytical  efforts  at  the  part  or  component  level  that  estimate  reliability  (or  reliability  improvement  if  comparing  with 
something  that  already  exists).  These  efforts  may  encompass  both  a  failure  mode  and  effects  analysis  (FMEA)  and 
failure  rate  calculations  for  each  of  the  failure  mechanisms.  Alternative  corrective  and/or  preventive  actions  that  could 
mitigate  the  most  significant  failure  mechanisms  should  be  identified. 

Additional  hardware  supporting  information 
for  technologies  used  to  protect  against  the  environment 

Analyses  of  how  the  environmental  protection  material  (e.g.,  coatings,  films,  composites,  and  so  forth)  will  be  formed 
or  integrated  chemically,  mechanically  (e.g.,  materials  and  fibers),  and/or  through  special  processes  (e.g.,  heat  treat¬ 
ment).  Basic  properties  characterized  potentially  through  virtual  modeling.  High-fidelity  simulations  conducted  to  pre¬ 
dict  behavior  in  the  field.  Predicted  behavior  meets  expectations/requirements. 

Software  TRL  4 
Definition 

Software  TRL  4 

Description 

Software  TRL  4 

Supporting  Information 

Module  and/or  sub¬ 
system  validation  in  a 
laboratory  environment 
(i.e.,  software  prototype 
development 
environment). 

Basic  software  components  are  integrated  to 
establish  that  they  will  work  together.  They  are 
relatively  primitive  with  regard  to  efficiency  and 
robustness  compared  with  the  eventual  system. 
Architecture  development  initiated  to  include  inter¬ 
operability,  reliability,  maintainability,  extensibility, 
scalability,  and  security  issues.  Emulation  with 
current/legacy  elements  as  appropriate.  Prototypes 
developed  to  demonstrate  different  aspects  of  the 
eventual  system. 

Advanced  technology  development, 
stand-alone  prototype  solving  a 
synthetic  full-scale  problem,  or 
standalone  prototype  processing 
fully  representative  data  sets. 

Additional  software  supporting  information  for  analysis  technologies 

An  initial  determination  of  the  analysis  input  data  (e.g.,  sensor  type  and  placement  and  historical  maintenance/failure 
rates)  has  been  made.  First-cut  rules  established.  These  rules  that  define  the  maintenance  actions  to  be  taken  as  a 
function  of  various  combinations  of  the  input  data. 

At  one  time,  the  FCS  TRA  identified  embedded  predictive  logistics  sensors  and  algorithms  as  a  CTE.  The  TRL 
assessment  in  the  actual  TRA  differed  from  this  example. 
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Table  2.  Additional  Life-Cycle-Related  Supporting  Information  for  TRL  5 


Hardware  TRL  5 
Definition 

Hardware  TRL  5 

Description 

Hardware  TRL  5 

Supporting  Information 

Component  and/ 
or  breadboard 
validation  in  a 
relevant 
environment. 

Fidelity  of  breadboard  technology 
increases  significantly.  The  basic 
technological  components  are  inte¬ 
grated  with  reasonably  realistic  sup¬ 
porting  elements  so  they  can  be 
tested  in  a  simulated  environment. 
Examples  include  “high-fidelity”  labo¬ 
ratory  integration  of  components. 

Results  from  testing  a  laboratory  breadboard  system 
are  integrated  with  other  supporting  elements  in  a 
simulated  operational  environment.  How  does  the 
“relevant  environment”  differ  from  the  expected  opera¬ 
tional  environment?  How  do  the  test  results  compare 
with  expectations?  What  problems,  if  any,  were 
encountered?  Was  the  breadboard  system  refined  to 
more  nearly  match  the  expected  system  goals? 

Additional  hardware  supporting  information 
for  technologies  to  improve  reliability/maintainability 

Where  warranted  by  risk,  accelerated  life-cycle  testing,  or  its  equivalent,  is  conducted  at  the  part  or  component  level 
in  high-stress  environments.  Results  from  analysis  of  test  results,  generally  measured  by  the  mean  time  between  fail¬ 
ure  (MTBF),  provide  estimates  of  product’s  life  and  performance  under  normal  conditions.  Reliability  growth  modeling 
indicates  future  levels  of  reliability  and  the  time  when  such  levels  will  be  achieved.  Estimates  meet  life-cycle  expecta¬ 
tions/requirements. 

Additional  hardware  supporting  information 
for  technologies  used  to  protect  against  the  environment 

For  material  formed  in  a  laboratory  environment,  basic  properties  have  been  verified  by  testing.  Variations  from  prop¬ 
erties  characterized  (perhaps  virtually)  at  TRL  4  are  acceptable.  Ability  to  use  the  material  for  the  intended  application 
in  terms  of  form,  fit,  and  function  has  been  verified. 

Software  TRL  5 
Definition 

Software  TRL  5 

Description 

Software  TRL  5 

Supporting  Information 

Module  and/or 
subsystem  vali¬ 
dation  in  a 
relevant 
environment. 

Level  at  which  software  technology  is 
ready  to  start  integration  with  existing 
systems.  The  prototype  implementa¬ 
tions  conform  to  target  environ¬ 
ment/interfaces.  Experiments  with 
realistic  problems.  Simulated  inter¬ 
faces  to  existing  systems.  System 
software  architecture  established. 
Algorithms  run  on  a  processor(s)  with 
characteristics  expected  in  the  opera¬ 
tional  environment. 

System  architecture  diagram  around  technology  ele¬ 
ment  with  critical  performance  requirements  defined. 
Processor  selection  analysis,  Simulation/Stimulation 
(Sim/Stim)  Laboratory  buildup  plan.  Software  placed 
under  configuration  management.  COTS/GOTS  (gov- 
ernment-off-the-shelf)  elements  in  the  system  software 
architecture  are  identified. 

Additional  software  supporting  information  for  analysis  technologies 

Test  data  are  developed  for  verifying  that  the  rules  established  at  TRL  4  are  working  as  intended.  Test  results  indicate 
adequate  performance.  Process  defined  for  integrating  the  output  of  on-board  analysis  technologies  with  other  off- 
board  information. 
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Table  3.  Additional  Life-Cycle-Related  Supporting  Information  for  TRL  6 


Hardware  TRL  6 
Definition 

Hardware  TRL  6 

Description 

Hardware  TRL  6 

Supporting  Information 

Syste  m/su  bsyste  m 
model  or  prototype 
demonstration  in  a 
relevant  environment. 

Representative  model  or  prototype  sys¬ 
tem,  which  is  well  beyond  that  of  TRL  5, 
is  tested  in  a  relevant  environment. 
Represents  a  major  step  up  in  a  technol¬ 
ogy’s  demonstrated  readiness.  Examples 
include  testing  a  prototype  in  a  high- 
fidelity  laboratory  environment  or  in  a 
simulated  operational  environment. 

Results  from  laboratory  testing  of  a  prototype 
system  that  is  near  the  desired  configuration 
in  terms  of  performance,  weight,  and  volume. 
How  did  the  test  environment  differ  from  the 
operational  environment?  Who  performed  the 
tests?  How  did  the  test  compare  with  expec¬ 
tations?  What  problems,  if  any,  were 
encountered?  What  are/were  the  plans, 
options,  or  actions  to  resolve  problems  before 
moving  to  the  next  level? 

Additional  hardware  supporting  information 
for  technologies  to  improve  reliability/maintainability 

Analytical  efforts  at  the  subsystem  level  that  estimate  reliability  (or  reliability  improvement  if  comparing  to  something 
that  already  exists).  Efforts  may  encompass  both  an  FMEA  and  failure-rate  calculations  for  each  of  the  subsystem 
failure  mechanisms.  Alternative  corrective  and/or  preventive  actions  that  could  mitigate  the  most  significant  failure 
mechanisms  should  be  identified.  Maintainability  analyses  conducted  to  determine  reliability-centered  (failure-based) 
maintenance  and  condition-based  maintenance  strategies  as  well  as  a  level  of  repair  determination.  Estimated  sup¬ 
port  man-hours  and  spare  parts’  needs  meet  expectations/requirements.  Form-fit-function  performance  ensured. 

Additional  hardware  supporting  information 
for  technologies  used  to  protect  against  the  environment 

Material  is  tested  in  a  laboratory  environment  to  provide  assurance  of  its  performance  throughout  its  intended  life 
cycle.  Deliberately  stressful/relevant  environments  are  used  to  determine  whether  any  degradation  in  performance 
occurs  against  known  standards.  Material  interaction  testing  is  conducted  to  ensure  that  no  adverse  chemical  or  other 
reactions  occur  in  either  the  components  being  protected  or  other  adjacent  parts  of  the  system. 

Software  TRL  6 
Definition 

Software  TRL  6 

Description 

Software  TRL  6 

Supporting  Information 

Module  and/or  subsys¬ 
tem  validation  in  a  rele¬ 
vant  end-to-end 
environment. 

Level  at  which  the  engineering  feasibility 
of  a  software  technology  is  demonstrated. 
This  level  extends  to  laboratory  prototype 
implementations  on  full-scale  realistic 
problems  in  which  the  software  technol¬ 
ogy  is  partially  integrated  with  existing 
hardware/software  systems. 

Results  from  laboratory  testing  of  a  prototype 
package  that  is  near  the  desired  configuration 
in  terms  of  performance,  including  physical, 
logical,  data,  and  security  interfaces.  Com¬ 
parisons  between  tested  environment  and 
operational  environment  analytically  under¬ 
stood.  Analysis  and  test  measurements 
quantifying  contribution  to  system-wide 
requirements  such  as  throughput,  scalability, 
and  reliability.  Analysis  of  human-computer 
(user  environment)  begun. 

Additional  software  supporting  information  for  analysis  technologies 

Verify  that  faults  can  be  detected/predicted  using  known  faults  in  a  simulated  real  environment,  such  as  a  test  cell  or 
test  platform  not  in  use.  Both  Type  1  errors  (actual  faults  not  detected)  and  Type  II  errors  (false  positives)  are  within 
acceptable  limits. 
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Table  4.  Additional  Life-Cycle-Related  Supporting  Information  for  TRL  7 


Hardware  TRL  7 
Definition 

Hardware  TRL  7 

Description 

Hardware  TRL  7 

Supporting  Information 

System  prototype 
demonstration  in  an 
operational 
environment. 

Prototype  near  or  at  planned  operational  system. 
Represents  a  major  step  up  from  TRL  6  by 
requiring  demonstration  of  an  actual  system  pro¬ 
totype  in  an  operational  environment  (e.g.,  in  an 
aircraft,  in  a  vehicle,  or  in  space).  Examples 
include  testing  the  prototype  in  a  test  bed  aircraft. 

Results  from  testing  a  prototype  system 
in  an  operational  environment.  Who 
performed  the  tests?  How  did  the  test 
compare  with  expectations?  What  prob¬ 
lems,  if  any,  were  encountered?  What 
are/were  the  plans,  options,  or  actions  to 
resolve  problems  before  moving  to  the 
next  level? 

Additional  hardware  supporting  information 
for  technologies  to  improve  reliability/maintainability 

Where  warranted  by  risk,  accelerated  life-cycle  testing,  or  its  equivalent,  is  conducted  at  the  subsystem-level  proto¬ 
type  in  high-stress  operational  environments.  Results  from  the  analysis  of  test  results,  generally  measured  by  MTBF, 
provide  estimates  of  a  product’s  life  and  performance  under  normal  conditions.  Reliability  growth  modeling  indicates 
future  levels  of  reliability  and  the  time  when  such  levels  will  be  achieved.  Estimates  meet  life-cycle  expectations/ 
requirements.  Refinement  of  maintainability  analyses  results  based  on  accelerated  life-cycle  testing.  Estimated  sup¬ 
port  man-hours  and  spare  parts’  needs  continue  to  meet  expectations/requirements. 

Additional  hardware  supporting  information 
for  technologies  used  to  protect  against  the  environment 

Operational  exposure  testing  conducted  on  multiple  subsystems  for  extended  periods  of  time  (concern  is  with  calen¬ 
dar  time,  notoperating  time).  Inspections  performed  throughout  the  testing  period.  Performance  measured,  in  part,  by 
man-hours  expended  or  scrap  rate.  Performance  is  verified  to  meet  expectations/requirements. 

Software  TRL  7 
Definition 

Software  TRL  7 

Description 

Software  TRL  7 

Supporting  Information 

System  prototype 
demonstration  in  an 
operational  high- 
fidelity  environment. 

Level  at  which  the  program  feasibility  of  a  soft¬ 
ware  technology  is  demonstrated.  This  level 
extends  to  operational  environment  prototype 
implementations  where  critical  technical  risk  func¬ 
tionality  is  available  for  demonstration  and  a  test 
in  which  the  software  technology  is  well  integrated 
with  operational  hardware/software  systems. 

Critical  technological  properties  are 
measured  against  requirements  in  a 
simulated  operational  environment. 

Additional  software  supporting  information  for  analysis  technologies 

Verify  that  faults  can  be  detected/predicted  using  known  faults  in  a  real  environment,  such  as  a  test  platform  not  in 
use.  Both  Type  1  errors  (actual  faults  not  detected)  and  Type  II  errors  (false  positives)  are  within  acceptable  limits. 
Process  for  integrating  the  output  of  on-board  analysis  technologies  with  other  off-board  information  is  verified. 
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Table  5.  Additional  Life-Cycle-Related  Supporting  Information  for  TRL  8 


Hardware  TRL  8 
Definition 

Hardware  TRL  8 

Description 

Hardware  TRL  8 

Supporting  Information 

Actual  system  com¬ 
pleted  and  qualified 
through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its 
final  form  and  under  expected  conditions.  In 
almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples 
include  developmental  test  and  evaluation 
of  the  system  in  its  intended  weapon  sys¬ 
tem  to  determine  if  it  meets  design 
specifications. 

Results  of  testing  the  system  in  its  final  con¬ 
figuration  under  the  expected  range  of  envi¬ 
ronmental  conditions  in  which  it  will  be 
expected  to  operate.  Assessment  of  whether  it 
will  meet  its  operational  requirements.  What 
problems,  if  any,  were  encountered?  What 
are/were  the  plans,  options,  or  actions  to 
resolve  problems  before  finalizing  the  design? 

Additional  hardware  supporting  information 
for  technologies  to  improve  reliability/maintainability 

Continuation  of  tests  conducted  to  measure  whether  the  technology  has  achieved  TRL  7,  failure  rates,  MTBF  growth 
charts,  maintenance  hours  per  operational  hour.  Operational  Test  and  Evaluation  (OT&E)  suitability  reports. 

Additional  hardware  supporting  information 
for  technologies  used  to  protect  against  the  environment 

Continuation  of  tests  conducted  to  measure  whether  the  technology  has  achieved  TRL  7.  OT&E  suitability  reports. 

Software  TRL  8 
Definition 

Software  TRL  8 

Description 

Software  TRL  8 

Supporting  Information 

Actual  system  com¬ 
pleted  and  qualified 
through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its 
final  form  and  under  expected  conditions.  In 
almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples 
include  developmental  test  and  evaluation 
of  the  system  in  its  intended  weapon  sys¬ 
tem  to  determine  if  it  meets  design  specifi¬ 
cations. 

Results  of  testing  the  system  in  its  final  con¬ 
figuration  under  the  expected  range  of  envi¬ 
ronmental  conditions  in  which  it  will  be 
expected  to  operate.  Assessment  of  whether  it 
will  meet  its  operational  requirements.  What 
problems,  if  any,  were  encountered?  What 
are/were  the  plans,  options,  or  actions  to 
resolve  problems  before  finalizing  the  design? 

Additional  software  supporting  information  for  analysis  technologies 

Verify  that  faults  can  be  detected/predicted  within  acceptable  limits  using  production-representative  platforms.  OT&E 
suitability  reports. 
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Table  6.  Additional  Life-Cycle-Related  Supporting  Information  for  TRL  9 


Hardware  TRL  9 
Definition 

Hardware  TRL  9 

Description 

Hardware  TRL  9 
Supporting  Information 

Actual  system 
proven  through  suc¬ 
cessful  mission 
operations. 

Actual  application  of  the  technology  in  its  final  form  and 
under  mission  conditions,  such  as  those  encountered  in 
OT&E.  Examples  include  using  the  system  under  opera¬ 
tional  mission  conditions. 

OT&E  reports. 

Additional  hardware  supporting  information 
for  technologies  to  improve  reliability/maintainability 

O&S  cost/failure  data  from  the  field. 

Additional  hardware  supporting  information 
for  technologies  used  to  protect  against  the  environment 

O&S  cost/failure  data  from  the  field. 

Software  TRL  9 
Definition 

Software  TRL  9 

Description 

Software  TRL  9 
Supporting  Information 

Actual  system 
proven  through  suc¬ 
cessful  mission- 
proven  operational 
capabilities. 

Level  at  which  a  software  technology  is  readily  repeatable 
and  reusable.  The  software  based  on  the  technology  is  fully 
integrated  with  operational  hardware/software  systems.  All 
software  documentation  verified.  Successful  operational 
experience.  Sustaining  software  engineering  support  in 
place.  Actual  system. 

Production  configuration  man¬ 
agement  reports.  Technology 
integrated  into  a  reuse  “wiz¬ 
ard”;  out-year  funding  estab¬ 
lished  for  support  activity. 

Additional  software  supporting  information  for  analysis  technologies 

Acceptability  of  Type  1  (actual  faults  not  detected)  and  Type  II  (false  positives)  errors  detected  in  the  field. 
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E.  Summary  and  Conclusion 

Life-cycle-related  technologies  affect  system  supportability  cost  and/or  time.  They  can 
reduce  the  logistics  footprint;  improve  reliability/maintainability;  lower  operating,  support,  or 
maintenance  manpower  requirements;  enhance  training;  enhance  human  factors  interactions; 
increase  operational  availability  or  readiness;  or  improve  the  ability  to  upgrade  the  system. 

While  life-cycle-related  technologies  can  be  considered  CTE  candidates  in  the  current 
TRA  process,  in  general,  they  receive  little  emphasis.  Increased  attention  on  such  technologies  is 
supported  by  the  CTE  definition,  the  policy  established  in  DoDI  5000.2  and  CJCSI  3170.01E, 
and  anticipated  increases  in  O&S  costs. 

To  improve  the  focus  on  life-cycle-related  technologies  during  CTE  identification, 
experts  in  the  life-cycle-related  areas  of  the  program  should  be  part  of  the  PM’s  technical  sup¬ 
port  team  that  makes  the  initial  CTE  determination  and  of  the  independent  review  team  (i.e.,  the 
SMEs),  which  can  suggest  changes.  Also,  when  deciding  whether  a  CTE  candidate  is  critical, 
additional  questions  should  be  asked  to  ascertain  whether  a  candidate  life-cycle  technology 
would  have  a  “significant”  affect  on  life-cycle  affordability: 

1.  Is  the  affordability  benefit  significant  over  the  life  cycle,  where  significance  is 
based  on  the  judgment  of  the  independent  SME  panel? 

2.  Is  the  affordability  benefit  enabled  by  a  technological  solution  and  not  solely 
through  engineering  design? 

These  questions  are  intended  to  highlight  whether  the  allocated  baseline  design  configuration 
contains  life-cycle-related  CTEs  and  must  be  answered  affirmatively. 

While  some  technologies  may  be  critical  solely  on  a  life-cycle  affordability  basis,  others 
may  be  critical  from  multiple  perspectives.  Consequently,  once  a  CTE  has  been  identified  from  a 
performance  perspective,  it  should  also  be  determined  whether  it  is  a  CTE  from  a  life-cycle  point 
of  view.  TRAs  evaluate  the  extent  to  which  a  program  is  ready  to  transition  to  the  next  phase  of 
development — an  evaluation  that  is  based  on  the  maturity  of  the  critical  technologies.  Therefore, 
if  a  technology  is  critical  from  both  perspectives  (perfonnance-related  maturity  and  life-cycle- 
related  maturity),  it  should  be  assessed  from  both  perspectives.  Establishing  performance-related 
maturity  is  not  a  sufficient  condition  for  assuming  life-cycle-related  maturity.  Separate  TRLs 
should  be  assigned. 

The  definitions,  descriptions,  and  supporting  information  corresponding  to  the  various 
TRLs,  as  described  in  the  TRA  Deskbook,  apply  to  life-cycle-related  technologies.  However, 
under  circumstances  in  which  the  long-term  effects  of  a  life-cycle-related  CTE  cannot  be  accu¬ 
rately  calculated  analytically  and  the  risk  of  a  miscalculation  is  large  (e.g.,  accelerated  life 
testing  of  explosives),  the  usual  supporting  information  for  TRL  4  and  above  should  be  aug¬ 
mented  or  tailored  to  the  specific  situation  to  help  clarify  the  basis  for  a  maturity  assessment. 
This  supplementary  supporting  infonnation  applies  to  (1)  hardware  TRL  data  for  technologies  to 
improve  reliability/maintainability  and  technologies  used  to  protect  against  the  environment  and 
(2)  software  TRL  data  for  analysis  technologies. 
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CTE  maturation  should  be  monitored  throughout  the  System  Development  and  Demon¬ 
stration  (SDD)  Phase  of  the  acquisition  framework.  All  CTEs  should  have  a  maturation  plan  that 
shows  a  roadmap  for  reaching  TRL  8  (actual  system  proven  through  successful  mission  opera¬ 
tions).  An  independent  technical  authority  should  monitor  the  status  of  CTE  maturation  plans 
during  systems  engineering  technical  reviews.  In  addition,  these  technical  reviews  will  be  the 
forum  for  identifying  any  new  life-cycle-related  CTEs  that  emerge  as  part  of  the  solution  to  a 
problem  encountered  during  system  development. 
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active  electronically  scanned  array 
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Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 
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commercial-off-the-shelf 
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critical  technology  element 
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Defense  Acquisition  Board 

DoD 

Department  of  Defense 

DoDI 

Department  of  Defense  Instruction 

DUSD(S&T) 

Deputy  Under  Secretary  of  Defense  for  Science  and  Technology 

FCS 

Future  Combat  Systems 

FMEA 

failure  mode  and  effects  analysis 

FY 

Fiscal  Year 

GOTS 

government-off-the-shelf 

IDA 

Institute  for  Defense  Analyses 

IMIS 

Integrated  Management  Information  System 
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Amphibious  Assault  Ship 

LHA-R 

Landing  Helicopter  Assault  Replacement 

LO 
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LRIP 

Low-Rate  Initial  Production 

MDA 

Milestone  Decision  Authority 

MDAP 

Major  Defense  Acquisition  Program 

MTBCF 

mean  time  between  critical  failures 

MTBF 

mean  time  between  failure 

NDAA 

National  Defense  Authorization  Act 

O&S 

operating  and  support 
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Office  of  the  Deputy  Under  Secretary  of  Defense  for  Science  and 
Technology 
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OT&E 

Operational  Test  and  Evaluation 
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Patriot  Advanced  Capability 
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PL 

Public  Law 

PM 

Program  Manager 

RERP 

Reliability  Enhancement  and  Re-Engining  Program 

SDD 

System  Development  and  Demonstration 

Sim/Stim 

Simulation/Stimulation 

SME 

subject  matter  expert 

SWL 

shipboard  weapons  loader 

TO 

technical  order 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 

U.S.C. 

United  States  Code 

WBS 

work  breakdown  structure 
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